STEREO-MEASUREMENT BORESCOPE WITH 3-D VIEWING 



FIELD OF THE INVENTION 

This invention relates generally to the field of borescopes and endoscopes, and 
more particularly to aborescope/endoscope which provides a 3-D image from a single 
camera and lens system. 

BACKGROUND OF THE INVENTION 

Various devices are known in the prior art for realizing a Ml color video 
picture of a target situated within a remote cavity. Most devices of this type employ 
an external light source conveyed to the image head by fiber optic bundles together 
with a solid state image sensor and lens system positioned in the distal end of the 
insertion tube of a borescope/endoscope, referred to herein as a probe, connected to an 
external video display. A particularly compact head including a light source and solid 
state image sensor lens system of this type is shown in U.S. Pat. No. 4,491,865 
(Danna et al.). 

Generally, in systems of this type, the fiber optic illumination bundle and the 
image sensor and optical system are disposed side by side in the end of a small 
insertion tube adapted to be inserted in cavities for viewing objects therein. The light 
provided by the fiber optic bundle has a field of view slightly displaced from the 
optical field of view of the image sensor, but generally overlapping sufficiently to 
provide an effective field of vision for the device. The image detected by the image 
sensor is displayed on a video screen and varies in magnification, apparent size, and 
detail, depending upon how close the end of the insertion tube carrying the lens 
system is from the object being viewed. Devices of this type typically have a depth of 
field from an eighth of an inch (3 mm) to something over four inches (100 mm). The 
displayed magnification decreases as the distance between the probe tip and the object 
being viewed increases. 

Attempts to measure an object in the image on the video display to determine 
the size of the object being viewed typically rely on either placing a known scale 
adjacent to the object to be measured for a comparison measurement, or providing a 
physical standoff over the lens on the end of the probe insertion tube, at which point 



the magnification is known and the end of the probe is adjusted until it just touches 
the object to be viewed at the standoff. With this known magnification, the image can 
be measured on the screen and the precise size determined. A related method uses 
optics having a very narrow depth of field and an adjustable focal point. Feedback 
from the focal point adjustment is used to determine the distance to the in- focus object 
and therefore the magnification of the object as viewed on the screen. This 
magnification is then used to perform measurements. 

Another measuring system is disclosed in U.S. Patent No. 4,980,763 (Lia) 
which measures objects viewed in a borescope by creating an auxiliary structure in the 
image, such as a shadow, which is projected onto the object so that its position in the 
video image changes in proportion to the distance of the image sensing head from the 
object. 

U.S. Patent No. 5,070,401 (Salvati et al.) discloses a 3-D video measurement 
system in which the depth or thickness of an object is determined along with its length 
and width. This system relies on the shadow method of the Lia patent to make its 3-D 
measurements. Although the method works well, it is difficult to achieve optimal 
shadow positioning and identification in some applications. 

Using stereo images for 3-D measurements is becoming popular. U.S. Patent 
No. 5,522,789 (Takahashi) discloses a stereo endoscope which includes a pair of 
objective optical systems, a pair of relay optical systems, an imagery optical system 
having a single optical axis, and a pair of imaging devices. U.S. Patent No. 5,860,912 
(Chiba) discloses a stereoscopic- vision endoscope system which uses two objective 
lens systems to provide separate images. The independent lens trains have diverging 
images so that, even at infinity, the views never overlap 100%. The separate lens 
trains also create the condition that the right image is displayed on the left side of the 
monitor and the left image on the right. The separate lens trains are typically very 
long, on the order of 15-20 mm for a 6 mm diameter probe, thus making the head of 
the probe very long. U.S. Patent No. 6,184,923 (Miyazaki) uses a plurality of optical 
systems in a first detachable tip adapter and a single optical system in a second 
detachable tip adapter with an endoscope having no optics adjacent to the imager to 
obtain multiple overlapping fields of view. This approach also typically yields a long 
distal-tip length and requires a complex attachment mechanism. 



SUMMARY OF THE INVENTION 

Briefly stated, two stereo images are created by splitting a single image into 
two images preferably using a field of view dividing splitter. The two images can be 
displayed side by side so that they can be viewed directly using stereopticon 
technology, heads-up display, or other 3-D display technology, or they can be 
separated for individual eye viewing. The two images focus on one imager such that 
the right image appears on the right side of the monitor and the left image appears on 
the left side of the monitor. The view of the images is aimed to converge at a given 
object distance such that the views overlap 100% at the object distance. Measurement 
is done with an onscreen cursor, a point matching process, and an optical data set. 

According to an embodiment of the invention, a device for viewing an object 
with a probe includes image splitting means for splitting an image of the object into 
first and second adjacent stereo image parts; image detecting means for detecting the 
stereo image parts; and focusing means for focusing the two stereo image parts from 
the image splitting means to the image detecting means; wherein the focusing means 
includes only one optical axis. 

According to an embodiment of the invention, a method for viewing an object 
with a probe includes the steps of (a) splitting an image of the object into first and 
second adjacent stereo image parts; (b) detecting the stereo image parts; and (c) 
focusing the two stereo image parts from the image splitting means to the image 
detecting means; wherein the step of focusing uses only one optical axis. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 A shows a block diagram of the optical system used in the invention. 
Fig. IB shows an optical path diagram used in explaining a first embodiment 
of the invention. 

Fig. 2 shows a video monitor operating with the first embodiment of the 
invention. 

Fig. 3 shows an optical path diagram used in explaining a second embodiment 
of the invention. 

Fig. 4 shows a video monitor operating with the second embodiment of the 
invention. 



Fig. 5 shows a stereo image in which a repeating pattern exists which creates a 
high probability of incorrect matches. 

Fig. 6 shows the two images of Fig. 5 superimposed on one another according 
to an embodiment of the present invention. 

Fig. 7 shows the two images of Fig. 5 with two matched point pairs. 

Fig. 8 shows part of an image used in explaining an embodiment of the present 
invention. 

Fig. 9 shows parts of two images used in explaining an embodiment of the 
present invention. 

Fig. 10 shows parts of two images used in explaining an embodiment of the 
present invention. 

Fig. 1 1 shows parts of two images used in explaining an embodiment of the 
present invention. 

Fig. 12 shows parts of two images used in explaining an embodiment of the 
present invention. 

Fig. 13 shows parts of two images used in explaining an embodiment of the 
present invention. 

Fig. 14 shows parts of two images used in explaining an embodiment of the 
present invention. 

Fig. 15 shows a borescope/endoscope system according to an embodiment of 
the invention. 

Fig. 16 shows an embodiment of a calibration tool used with an embodiment 
of the present invention. 

Fig. 17 shows a cross-section taken along the line 17-17 in Fig. 16. 

Fig. 18 shows a base of the calibration tool of Fig. 16 with a plurality of grid 
lines arrayed across an upper surface of the base. 

Fig. 19 shows left and right stereo images taken of the base of Fig. 18 through 
the calibration tool of Fig. 16. 

Fig. 20A shows a stereo image of part of an object used in explaining a feature 
of the invention. 

Fig. 20B shows a stereo image of another part of the object partially shown in 
Fig. 20B. 



Fig. 21 shows a functional connection of the parts of an embodiment of the 
stereo borescope/endoscope system. 

Fig. 22 A shows a stereo image with a pair of horizontal lines on which 
individual point matching is very difficult or impossible. 

Fig. 22B shows a stereo image with a grid which creates a high probability of 
incorrect point matching. 

Fig. 23 A shows the image of Fig. 22 A with the addition of a shadow line. 

Fig. 23B shows the image of Fig. 22B with the addition of a shadow line. 

Fig. 24 shows the relationship of left/right shift vs. object distance. 

Fig. 25 shows the object-distance error caused by a unit error in left/right shift. 

Fig. 26 shows how the reflection of the illumination fiber bundle changes 
position based on the distance to the reflective surface. 

Fig. 27 shows the use of small zoomed views concurrently with the unzoomed 
stereo image. 

Fig. 28 shows the use of a dynamic icon to help the user step through the 
measurement process. 

Fig. 29 shows the use of a dynamic icon to help the user step through the 
measurement process. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Referring to Fig. 1 A, a probe 46 contains an imager 12. An optical system 44 
includes an image splitter 1 1 and a single axis focuser 17. Image splitter 1 1 divides 
the field of view of focuser 17 into two overlapping fields of view which have 
different optical axes. The two fields of view are then focused on imager 12 by 
focuser 17. 

Referring to Figs. 1B-2, an embodiment of optical system 44 shows an object 
10 imaged onto imager 12 through a prism 14 and a lens assembly 16. The image of 
object 10 is split into two stereo image parts 18, 19 by prism 14. Prism 14 is 
preferably a refractive image-splitting prism, such as, for example, a wedge prism. 
Image parts 1 8 and 19 are then displayed on a monitor 20. The geometric dimensions 
of object 10 are then measured using at least one onscreen cursor along with a 



measurement process as discussed below. A transparent window 15 is optionally 
used to prevent fluids from contacting prism 14. 

Using a single prism 14 to split the image rather than using two separate lens 
assemblies simplifies and shortens the length of the optical assembly, while 
preventing the overlap of image parts 18 and 19 on imager 12 without the use of 
complex masking. Image parts 18, 19 preferably focus on imager 12 such that a right 
image 22 appears on the right side of monitor 20 and a left image 21 appears on the 
left side of monitor 20 without electronic manipulation. The two fields of view 
created by the splitting prism are aimed to converge at a given object distance such 
that the views overlap 100% at the object distance. Measurements can only be 
performed where the views overlap. It is therefore desirable to maximize the overlap 
within the measurable object-distance range, which is typically from about 5 mm to 
about 25 mm for a 4 mm to 10 mm diameter probe. By making the fields of view 
overlap 100% within this range, a larger portion of the image can be used for 
measurements than if the views overlap 100% outside this range. In the case of 
parallel or diverging optical axes, the two fields of view never overlap 100%, which 
makes a large portion of the image unusable for measurements. Mirrors could be used 
in place of prism 14. Although using mirrors reduces the compression and bending 
aberrations introduced by prism 14, achieving the same optical separation using 
mirrors requires a much larger optical assembly than does the equivalent prism. Thus, 
prism 14 is more desirable for small-diameter probes. 

Images 18, 19 are preferably displayed side by side on a video monitor 20 as 
left and right images 21, 22 for direct 2-D viewing. The images can also be viewed 
using stereopticon technology, heads-up display, or other 3-D display technology to 
convey the depth information captured in the stereo image. A single image is 
optionally displayed, because positioning a probe using the live image with both 
stereo image parts displayed can be distracting and fatiguing. Displaying only one of 
the stereo image parts during positioning eliminates the distraction caused by the 
second image part, thus reducing fatigue. 

The index of refraction of the medium contacting the wedged face of prism 14 
affects the convergence angle and effective separation of the two fields of view. If the 
medium has the same index of refraction as the prism material, image splitting does 



not occur. By adding optional transparent window 15, contact between prism 14 and 
external media, such as fluids, is prevented thus ensuring that splitting occurs in the 
desired medium, typically air, and convergence is maintained. The effects of external 
media between window 15 and object 10 are then limited to increasing the object 
distance of the 100% overlap point and reducing fields of view. 

Referring to Figs. 3-4, an alternative embodiment is shown in which stereo 
optical system 44 includes a non-symmetrical prism 14' on which a ridge 13 is 
displaced from the center of the optical axis of single-axis lens 16 such that one of the 
stereo views, the "main image path" in Fig. 3, is imaged onto more than one half of 
imager 12, while the other stereo view, the "second image path" in Fig. 3, is imaged 
onto less than one half of imager 12. The magnification and convergence angle of the 
second image path are controlled by the contour and angle of prism surface 14a. The 
image provided by the "main image path" is preferably displayed on a monitor 30. 

The image provided by the second optical path is used by a computational 
element to obtain a correlation with the image from the first optical path. This 
correlated image is used by the computational element, along with data describing the 
characteristics of the optical system, to determine dimensional parameters of viewed 
objects and provide measurement results to a display, such as video monitor 30 and/or 
memory/file/storage, preferably accessible to the user. This system can also be 
provided with more than two optical paths, such that intersection of the optics for the 
stereo measurement can be specified for near and far measurement in the same 
system. The asymmetrical splitting of the image can also be performed with a non- 
prism based splitter such as a dual lens assembly. The computational element can 
provide a probability estimate for a successful image to image correlation, and 
optionally request user input to validate or correct the correlation if the probability 
falls below a preset level. 

The invention provides an automatic system for measuring various parameters 
of an object being viewed by the above described stereo optical imaging systems 
without the need for a known scale or optical reference, such as a shadow or laser 
projection. Using this system, three dimensional measurements can be made of 
objects viewed by a single, two dimensional image sensor. Measurements are 
preferably corrected for magnification (left and right), lens distortion, and other 



factors which would otherwise degrade the accuracy of the measurement results 
produced by the above mentioned computational element. 

A number of systems exist which acquire images from two or three different 
perspectives. The relative positions of the objects in the different images change 
based on the distance between the camera and the objects. This change in relative 
position allows measurements to be performed. One of the most challenging aspects 
of such systems is identifying matching points in the different images. Many 
algorithms have been developed to create disparity maps from these images which 
indicate the distance to points in the images. These algorithms are generally 
computationally intensive and provide more information than is needed for 
applications that require only two to four point pairs. Other systems search the 
images for a good match without any consideration for the global alignment of the 
images. This often results in incorrect matches when there are repeating patterns or 
multiple similar points in the images. 

The present invention performs a relatively fast, automatic, global alignment 
prior to matching specific points for measurement. The global alignment is then used 
to limit the search range on the point-pair matching, thus improving the speed and 
reliability of the matching process. 

Referring to Fig. 5, a stereo image is shown in which a repeating pattern exists 
which creates a high probability of incorrect matches when global alignment is not 
taken into consideration. Prior to performing specific point matching, a global 
alignment is performed. The global alignment consists of obtaining global vertical 
shift data and regional horizontal shift data. The vertical shift between the left and 
right images, referred to as the global vertical shift, is caused mainly by optical 
misalignments and is relatively constant throughout the image. The horizontal shift 
between the left and right images is dependent on the optical characteristics and the 
distance from the distal tip of the probe to the surface being viewed, which may vary 
throughout the image depending on the contour of the surface and the tip orientation. 
The regional horizontal shift data indicates the horizontal shift at one or more 
particular positions within the image. 

Referring to Fig. 6, the global alignment has been used to create a 
superimposed view of the two images to the right of the image of Fig. 5. When 



specific points are then matched, the search area can be reduced based on the global 
alignment information, thus speeding up the search and reducing the likelihood of 
incorrect matches. Fig. 7 shows the image with two matched point pairs. 

There are two main processes that this approach employs. The first is the 
global alignment process which is preferably performed as follows. 

1) Convert the image to black and white and decimate the image to reduce 
processing time (optional). 

2) Perform edge detection on the left image as shown in Fig. 8. 

3) Find multiple edge endpoints or non-straight edge points which are 
relatively easy to match using simple matching algorithms, as shown by the crosses in 
the left image of Fig. 9. 

4) Find matching points in the right image using a standard algorithm such as 
simple pixel value comparison, shown in Fig. 10. 

5) Count the number of times each possible left image to right image vertical 
shift occurs in the set of matched point pairs. 

6) Find the most common, or global, vertical shift. The vertical shift is 
essentially constant regardless of object distance or position in the image. 

7) Attempt to realign points that do not conform well to the global vertical 

shift. 

8) Throw out points that appear to be invalid. The remaining "good" matches 
are shown as the crosses in Fig. 11. The horizontal shifts between these matched 
point pairs provide the regional horizontal shift data. 

The second step is the matching of the specific points being used for the 
measurement, preferably performed as follows. 

1) The user identifies a desired point in the left image as shown by the cross 
in Fig. 12. 

2) Nearby points are found that were used in the global alignment as shown 
by the crosses in Fig. 13. 

3) The maximum and minimum horizontal shifts of the nearby points are 
found. The horizontal shift may vary with position depending on the orientation and 
contour of the target surface. 



4) The right image is searched for a match using the above maximum and 
minimum horizontal shifts along with the global vertical shift to limit the search 
region. The result is shown in Fig. 14. 

The matching process is preferably implemented in software, but 
implementation in hardware or firmware is possible. The matching process can also 
be implemented manually or as a combination of manual steps and automatic steps. 
In a purely manual mode, the user identifies the matched point in both images. If the 
system cannot automatically find points having only one good match for use in the 
global alignment step, the user is optionally asked to manually identify a matched 
point to give the system a starting point for the global alignment. Additionally, if the 
automatic matching process finds either multiple good matches or no good match for 
a specific user-identified point, the user is optionally asked to identify the correct 
match. 

Referring to Fig. 15, a borescope/endoscope system 40 according to an 
embodiment of the invention is shown. A detachable distal tip 42 contains optical 
system 44. Imager 12 is included in a probe 46. An optical data set 70 describing 
optical system 44 is generated using either factory calibration 72, field calibration 74, 
or a fixed calibration table 76. Optical data set 70 is preferably stored in non- volatile 
memory in probe electronics 48 and passed to a CPU 56 for use in stereo 
measurement. This approach allows probe 46 to be used with different processor 
boxes without requiring a manual transfer of optical data set 70. Probe electronics 48 
also convert the signals from imager 12 to a format accepted by a video decoder 55. 
Video decoder 55 produces a digitized version of the stereo image produced by probe 
electronics 48. A video processor 50 stores the digitized stereo image in a video 
memory 52, while giving CPU 56 access to the digitized stereo image. CPU 56, 
which preferably uses both a non-volatile memory 60 and a program memory 58, 
performs the global alignment, point matching, and measurement using the digitized 
stereo image and optical data set 70. A keypad 62, a joystick 64, and a computer I/O 
interface 66 preferably convey user input for such functions as cursor movement to 
CPU 56. Video Processor 50 superimposes graphics such as cursors and results on 
the digitized image as instructed by CPU 56. An encoder 54 converts the digitized 
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image and superimposed graphics into a video format compatible with monitor 20 on 
which left image 21, right image 22, and superimposed graphics are displayed. 

As precise as manufacturing optics elements has become, every optical device 
is different. Every optical system 44 is preferably calibrated in conjunction with 
probe 46, especially when optical system 44 resides in detachable distal tip 42. 
Optical data set 70 is preferably determined for the probe and optical system 
combination. Three calibration methods are preferably used to generate optical data 
set 70. Factory calibration 72 is performed when the probe unit is originally 
manufactured, as well as when a customer sends the unit back to the factory for 
calibration. Field calibration 74 is performed outside the factory preferably whenever 
a new detachable distal tip 42 is placed onto probe 46. Table calibration using a 
calibration table 76 is optionally used for certain applications. If one assumes a 
certain similarity of lenses due to very precise manufacturing technologies, one probe 
and optical system is calibrated, and calibration table 76 is made from the results. 
Other probes then use the same calibration table 76. Alternatively, calibration table 
76 is optionally generated through theoretical analysis of the optical system. For 
some applications, using calibration table 76 provides adequate calibration accuracy. 

Referring to Figs. 16-17, a calibration tool 80 is shown. Calibration tool 80 
includes a base 82 and a cover 84. Cover 84 preferably fits over base 82 such that 
part of cover 84 fits against an upper surface 86 of base 82. Base 82 includes a post 
88 with a line 94 on top of post 88. A cavity 92 is shaped to receive detachable distal 
tip 42 attached to probe 46, or in the case of a non-detachable design, the tip of probe 
46 containing optical system 44. 

Referring to Fig. 18, a plurality of grid lines 90 of known spacing d3 are on 
upper surface 86 of base 82. The length of line 94 is a known distance dl, while the 
height of post 88 is a known distance d2. Fig. 19 shows the view of left and right 
images 21, 22 of post 88 and surface 86 as displayed on monitor 20 (Fig. 15). Note 
how grid lines 90 become distorted the further away from post 88 they are. This 
distortion is known as optical mapping distortion. 

Optical data set 70 preferably includes calibration parameters pertaining to (a) 
optical mapping distortion, (b) magnification at a particular object distance, and (c) 
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left to right image shift vs. object distance, i.e., parallax information. These 
calibration parameters can be determined using calibration tool 80. 

In the preferred embodiment, a calibration process executed by CPU 56 
automatically locates gridlines 90. Equations are generated which characterize the 
grid lines. These equations are stored in optical data set 70. Compensation equations 
are then generated which are used to correct the optical mapping distortion giving a 
grid with equally-sized squares regardless of position in the image. The process then 
automatically identifies line 94. The distortion correction equations are used to 
correct the positions of the endpoints of line 94. Given the distortion-corrected length 
of dl, the distortion-corrected grid spacing d3, and known height d2, the actual object 
distance from the effective optical origin to surface 86 is computed. This object 
distance and the absolute magnification determined from known spacing d3 are stored 
in optical data set 70. Optionally, if a single point on the top of post 88 is used in 
conjunction with distances d2 and d3, as long as either the tip to post or tip to grid 
distance is known. 

Finally, the left to right image shifts between the distortion-corrected gridlines 
on surface 86 and the distortion-corrected endpoints of line dl are used in conjunction 
with the computed object distances to surface 86 and the top of post 88 to compute the 
left to right image shift vs. distance. This parameter is then stored in optical data set 
70. It is also possible to store the positions of the automatically identified points in 
optical data set 70 and compute the relevant optical parameters from those positions 
when measurements are performed. 

Definition of variables: 

NearOD == object distance to line 94 
Far_OD = object distance to gridlines 90 

dl_pixels = distortion-corrected length of line 94 in image in pixels 

d3_pixels = distortion corrected gridline spacing in image in pixels 

Close_shift = difference in the distortion corrected horizontal positions of points on 

line 94 as they appear in the left and right image parts in pixels, i.e., horizontal shift. 
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Far_shift = difference in the distortion corrected horizontal positions of points on 
gridlines 90 between the left and right image parts in pixels, i.e., horizontal shift. 
P = parallax constant 

Far_OD/Near_OD = (dl_pixels/dl) / (d3_pixels/d3) 
Far_OD =Near_OD * (dl_pixels/dl) / (d3_pixels/d3) 
Far_OD = Near_OD + 62 

Near_OD *((dl_j)ixels/dl) / (d3_pixels/d3) - 1) = d2 
NearJDD = d2/((dl_pixels/dl) / (d3_pixeis/d3) - 1) 
Far__OD = Near_OD + 62 

P = Near_OD * ((Far_shift - Nearjshift) / 2) / (Near__OD/Far_OD - 1) 

The magnification in pixels/unit length and object distance at either line 94 or 
gridlines 90 are saved as the magnification at a particular distance. 

This method of calibration offers a number of advantages over traditional 
methods. Computing the object distances to the surfaces is preferred over using the 
by-design or typical object distances because it eliminates errors caused by fixture and 
optical system manufacturing tolerances as well as imperfect seating of the tip in 
cavity 92. Optical mapping distortion is directly measured rather than relying on 
by-design or typical values. This is especially important given the complexity of the 
distortions generated by wide field-of-view lens system 16 and prism 14 having steep 
surface angles combined with mechanical alignment tolerances. Determining all 
parameters from a single captured image rather than, for example, one image for 
distortion mapping and another for magnification and parallax, eliminates errors that 
are induced by capturing the two images from slightly different viewing angles or 
with the detachable tip position shifted slightly relative to the optics in the probe tip. 
Combined with automatic point and line identification, using a single fixture and 
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image also makes the calibration process simple and inexpensive enough to allow 
users to perform the calibration with new detachable tips themselves without having 
to return the probe to the factory. 

In an alternative embodiment, a single point on the top of post 88 is used in 
conjunction with distances d3 and known Near OD or Far OD. In this case, P may 
be computed directly using these dimensions and the distortion corrected pixel 
locations of the point on post 88 and gridlines 90. This approach provides all the data 
necessary to perform measurements, but mechanical positioning of the optical system 
relative to the target points is more critical than with the preferred embodiment. 

In probe systems having multiple, individually calibrated, detachable 
measurement tips, the system must know which tip is currently attached so that the 
correct optical data set 70 is used for measurements. The user typically must indicate 
to the system which tip is being used, and it is not unusual for the user to identify the 
wrong tip. Because different tips have different optical characteristics, the image 
being used for measurement can be analyzed to be sure it matches the tip that has 
been identified. With stereo tips, there is typically a vertical offset between the left 
and right image parts caused by accidental, or possibly intentional, differences in the 
positioning of optics in the left and right image paths. If the offset detected in the 
measurement image is different from the offset detected during calibration, the user is 
optionally notified that the incorrect tip may have been identified. Furthermore, the 
offset is optionally used to automatically identify the tip being used. 

Measuring an object using the calibrated system 40 is preferably performed 
using the following steps. 

1) Using onscreen cursors and user input through joystick 64 and/or keypad 
62, identify points in the left image relevant to the measurement. 

2) Identify matching points in the right image automatically or manually. 

3) Un-distort both the left and right side positions of the identified points 
using the optical mapping distortion parameters. 

4) Find the left/right unit shift for each point pair using the 
distortion-corrected left and right side point positions. 

5) Using the results of step (4), and the shift vs. distance parameter, compute 
the object distance at each point: 
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defining Point_OD = object distance to a point on an object and 
Point_shift = difference in the distortion corrected horizontal positions of the 

point as it appears in the left and right image parts in pixels, i.e. horizontal shift, we 

have 

PointjDD = P / ( P / FarJDD - (Far^shift - Point_j3hift)/2). 

6) Using the results of steps (3) and (5) and the magnification at a given 
distance parameter, compute the unit magnification at each point: 

definingPoint Mag = magnification in pixels/unit length at the point, we have 
Point JVIag = d3_pixels/d3 * Far_OD / Point_OD 

7) Using the 2-D unit spacing between points, the unit magnifications at each 
point, and the object distance for each point, compute the 3-D distance between 
points, using either the left, right, or both images to do so. 

This basic process can be extended to include more than two points in various 
combinations to measure more complex geometries such as areas, volumes, distances 
between combinations of points, lines, and planes, angles between lines and/or planes 
and the like. 

The unit spacing is preferably in terms of pixels, while the unit magnification 
is preferably in terms of mm per pixel or inches per pixel, although other units may be 
used. The measurement process is preferably carried out in software residing in 
program memory 58 (Fig. 15), but is optionally implemented in hardware or 
firmware. 

Referring to Figs. 20A-20B, when an object spans more than one field of 
view, two or more stereo images are used to perform measurements. An object 101 is 
shown which extends from a point 103 (Fig. 20A) to a point 102 (Fig. 20B). A point 
104 is an intermediate point between point 103 and point 102 which is visible in the 
left and right images of both Fig. 20A and 20B. The distance from point 104 to point 
103 (Fig. 20A) is measured and added to the distance from point 104 to point 102 
(Fig. 20B) to obtain the total distance for object 101. The procedure is preferably 
carried out using as many different views as necessary to obtain the complete 
geometric characteristics of the object. In the preferred embodiment, the user 
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identifies only end points 102 and 103. The system then automatically identifies 
intermediate points, such as point 104, in each of the images. 

Referring to Fig. 21, a functional connection of the parts of an embodiment of 
the stereo borescope/endoscope system is shown. A stereo optical system 202, 
optionally implemented in a detachable tip, is combined with a probe 204. A 
measurement means 206 receives stereo images from probe 204 and begins the 
measurement process. The measurement means preferably uses a matching means 
208 to match points on the left and right stereo images. The matching is done either 
manually (210), automatically (212), or with a combination of manual and automatic 
techniques. The measurement means also preferably uses an optical data set 214 
which is generated by factory calibration 216, field calibration 18, or a calibration 
table 220. Measurement means 206 then produces outputs 222 which preferably 
includes a measurement result, 3-D data about the object, and information concerning 
the accuracy of the measurements. 

It is often desirable to transfer measurement images from 
borescope/endoscope system 40 to a computer for re-measurement, possibly by an 
individual with more expertise than the user who captured the image. To perform 
re-measurement, it is necessary to have both the image and optical data set 70. Saving 
the image and the optical data set in separate files creates a risk that the wrong optical 
data set will be used for re-measurement or that the two files will become separated 
making re-measurement impossible. The image and optical data set, possibly along 
with data pertaining to any measurements that have already been performed, are 
preferably saved in a single file for later use as described in U.S. Provisional 
Application Ser. No. 60/270,967 filed February 22, 2001 and entitled METHOD FOR 
STORING CALIBRATION DATA WITHIN IMAGE FILES, incorporated herein by 
reference. This approach assures that the correct optical data set is used for 
re-measurement and prevents the separation of the optical data set from the image 
data. 

In another embodiment of the invention, a plurality of points in a given area, 
i.e., the stereo left/right pairs, are assembled and three dimensional information 
derived from the points is structured into a finished file which permits reconstructing 
at least one geometric characteristic of the image. Such a file could be a 3D CAD file, 
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contour map file, grid file, or a wire frame file. The more complicated the surface 
being viewed, the more points required to produce an accurate model. Geometric 
characteristics include such items as length, area, volume, angle, and the like. 

In another embodiment of the invention, the measuring process includes 
projecting a pattern from an off-imaging axis onto the object being viewed such that 
the pattern tracks across the object relative to a distance of the object from the device, 
and using a location of the pattern on the object to aid determination of the 
measurement data. In some cases, the projected pattern can provide a unique feature 
to improve the accuracy of the point matching process. Fig. 22A shows a stereo 
image of a pair of horizontal lines. Matching individual left/right points on the lines 
is difficult or impossible due to the lack of unique details along the lines. Fig 22B 
shows a stereo image of a grid. The repeating pattern of the grid creates a high risk of 
incorrect left/right point matching. In Fig. 23A and Fig. 23B, projected shadow lines 
120 and 121 are added to the images. In Fig. 23 A, the points where shadow line 120 
crosses the horizontal lines can now be accurately matched allowing the line spacing 
to be measured. In Fig. 23B, assuming a limited global vertical shift, alignment 
of the gridlines on the left and right sides is made clear, greatly reducing the risk of 
incorrect point matching. These embodiments are enabled by the disclosures in U.S. 
Patent No. 4,980,763 (Lia) and U.S. Patent No. 5,070,401 (Salvati et al.), both of 
which are incorporated herein by reference. These embodiments are preferably used 
in conjunction with the measuring process described above to enhance the process. 
On many surfaces, a reflection of the light fiber bundle from the surface can be 
detected. As shown in Fig. 26, the position of this reflection in the left and right 
images varies with the object target distance, so the reflection could also be used in 
the stereo measurement process. 

In another embodiment of the invention, multiple images of an object to be 
measured are captured from different perspectives. The user then identifies the points 
relevant to the measurement in one image. The point matching process is used to find 
automatically the same left and right side points in the other captured images, with the 
result being computed from each image. The multiple results are then combined to 
produce a final result that is likely to be more accurate than any of the individual 
measurements. 
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The accuracy of the measurements performed is dependent on several factors 
including object distance, image resolution, distortion correction accuracy, and point 
matching accuracy. As shown in Fig. 24, the left/right shift changes considerably 
with a unit change in object distance at close object distances. At larger object 
distances, the left/right shift changes much less for the same unit change in object 
distance. This means that the impact of a 1 unit error in left/right shift, caused by 
imperfect distortion correction and/or matching, increases with object distance. This 
increase is exponential in nature as illustrated in Fig. 25. A combination of computed 
object distance, image resolution, point matching sureness, and typical distortion 
correction accuracy is preferably used to estimate the unit accuracy tolerance of the 
measurement. This unit tolerance combined with the measurement result provides a 
percent accuracy tolerance. Either of these measures or another measure derived from 
these measures is preferably displayed to the user. Allowing the user to set a 
maximum tolerance limit, above which a visual or audible notification is made, 
reduces the likelihood that an unacceptably-large tolerance will go unnoticed. 

In another embodiment of the invention, the object distance, as determined by 
the measurement process, is used to select a point-spread function (PSF) which 
accurately characterizes the blurring effects of the optical system when viewing 
objects at that object distance. Deconvolution techniques are then used to reverse 
these blurring effects yielding a more in-focus image than otherwise. 

Referring to Fig. 27, accurate placement of the measurement cursors and 
matching of the identified points are critical to achieving measurement accuracy. It 
can be difficult, especially with small monitors, to make accurate cursor placements 
and to verify accurate matching. One solution is to zoom the entire image so that the 
area around the cursor is seen more easily. This approach, however, typically hides a 
portion of the image. If a repeating pattern exists, it is sometimes impossible to tell 
where in that pattern the cursor is positioned. Fig. 27 shows the concurrent display of 
the main left and right images along with smaller zoomed windows showing the left 
and right cursor positions in much greater detail. In this way, the position within 
repeating patterns is still visible while making accurate cursor placement and 
matching verification much easier. Alternatively, the area surrounding only the user- 
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designated cursor is optionally shown in a smaller zoomed window to reduce 
processing time and/or display area usage. 

It is important for the user to verify that the automatic point matching function 
properly identifies the match in the right image. In other systems, the user must 
position a cursor in the left image and then press a button before the automatic 
matching of the point begins. The system will then typically go on to the placement 
of the next cursor in the measurement. If the match is incorrect, the user must then 
take other steps to either correct the match or to select and move the left cursor to see 
if the system can match a slightly different point. 

According to the present invention, a preferred method is to start the automatic 
match as soon as movement of the left cursor stops. This way, the user can see if the 
match is correct without any other input. If the match is incorrect, the user can 
immediately move the cursor again to see if a correct match is achieved with a slight 
change in position without performing additional steps to activate the cursor that was 
improperly matched. With a fast enough matching function, the match is performed 
in real-time as the left cursor is moved, so the user can immediately see if the match is 
correct. 

Referring to Figs. 28-29, multiple points must be identified to perform a 
measurement. When measurements require more than two points, such as in point to 
line or point to plane measurements, the user must know whether the cursor being 
placed is used to define the "point", the "line", or the "plane". On-screen instructions 
tend to be very consuming of display area. Using a single icon which indicates both 
the type of measurement and how the cursor being placed is used in the measurement 
allows the same display space to provide both informational items. In Fig. 28, the 
icon in the upper right corner, and in particular the cross symbol, shows that a point to 
plane measurement is being performed and that the active cursor is used in the 
definition of the plane. In Fig. 29, the icon has changed, with the cross symbol 
showing that the active cursor defines the off-plane point. 

When more than one type of measurement is available, it is possible that the 
user will place one or more cursors before deciding that they would rather perform a 
different type of measurement. In other systems, changing the measurement type 
causes the cursors that have been placed to be removed. Often, however, the points 
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that were already placed could have been used with the new measurement type. The 
user then has to repeat the placement of the lost cursors. According to the present 
invention, a preferred method is to keep the positions of any usable cursors that have 
already been placed when the measurement type is changed. 

While the present invention has been described with reference to a particular 
preferred embodiment and the accompanying drawings, it will be understood by those 
skilled in the art that the invention is not limited to the preferred embodiment and that 
various modifications and the like could be made thereto without departing from the 
scope of the invention as defined in the following claims. 
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